perf(ggml/hip): fp3 VDR=4 rewrite + fp2 ILP/ds_swizzle for ROCMFPX MMVQ#20
Closed
DeanoC wants to merge 1 commit into
Closed
perf(ggml/hip): fp3 VDR=4 rewrite + fp2 ILP/ds_swizzle for ROCMFPX MMVQ#20DeanoC wants to merge 1 commit into
DeanoC wants to merge 1 commit into
Conversation
… ROCMFPX MMVQ Two RDNA-only MMVQ optimizations for the ROCMFPX quant family, stacked on the fp3 matvec patch (828186a). fp3 (Q3_0_ROCMFPX): restructure vec_dot_rocmfpx_fp3_q8_1 from VDR=2 to VDR=4, half-block aligned. With VDR=4 the MMVQ launcher dispatches iqs in {0,4}, so each vec_dot covers exactly one 16-weight half-block (4 groups) with a single scale and no per-element straddle branch: the 48-bit weight window is spliced out in one 64-bit read and fed through two independent dp4a chains. Doubles work-per-lane, halves lanes-per-block and the K-loop trip count. fp2 (Q2_0_ROCMFP2): split vec_dot_rocmfpx_fp2_q8_1 into two independent dp4a chains (byte-identical - integer accumulation is associative and the lane assignment is unchanged) and extend the ds_swizzle warp reduction (previously fp3-only) to fp2. Shared helper renamed warp_reduce_sum_rocmfpx_dsswizzle. Methodology (lucebox3, gfx1151 Strix Halo, ROCm 7.2.4; DSpark spec decode, verify-width 4, mclk-pinned 1000 MHz, 2 warmups + 7 reps, matched-clock): build spec tok/s vs base accept correctness base (828186a) 24.3 -- 0.78 byte-identical to stock fp2 only 24.5 +1.0% 0.78 byte-identical (SHA match) fp3 VDR=4 26.6 +9.0% 0.79 verified correct (*) fp3 + fp2 (this) 26.6 +9.0% 0.79 verified correct (*) fp3-isolated kernel time, type-104 vs the unchanged type-101 fp4-fast control (clock-independent ratio): -18.5% (0.379 -> 0.309). (*) fp3 VDR=4 is NOT byte-identical: regrouping the per-lane accumulation reassociates the cross-lane float reduction (<=~1 ULP per block, far below the argmax margin). Validated instead by: greedy math (=630) and 3-pet logic correct, structured prompts (JSON / planet facts / number spelling) byte-identical to base, and spec accept_rate preserved (0.78 -> 0.79). fp2 is fully byte-identical; its +1% is absorbed in the combined build (26.6 == fp3-alone) and is included as a zero-risk companion cleanup. Quality suite: 8 diverse greedy prompts (code, algorithms, exact math, logic, long-form, JSON, facts, repetition) - no regressions.
DeanoC
marked this pull request as draft
July 16, 2026 12:12
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two RDNA-only MMVQ optimizations for the ROCMFPX quant family, stacked on #19 (
828186a). Merge #19 first — this PR is based on its head branch.Q3_0_ROCMFPX) — restructurevec_dot_rocmfpx_fp3_q8_1fromVDR=2toVDR=4, half-block aligned. AtVDR=4the MMVQ launcher dispatchesiqs ∈ {0,4}, so eachvec_dotcovers exactly one 16-weight half-block (4 groups) with a single scale and no per-element straddle branch. The 48-bit weight window is spliced out in one 64-bit read and fed through two independent dp4a chains. Doubles work-per-lane, halves lanes-per-block and the K-loop trip count.Q2_0_ROCMFP2) — splitvec_dot_rocmfpx_fp2_q8_1into two independent dp4a chains and extend theds_swizzlewarp reduction (previously fp3-only) to fp2. Shared helper renamedwarp_reduce_sum_rocmfpx_dsswizzle.Methodology
lucebox3 (gfx1151 Strix Halo, ROCm 7.2.4). DSpark spec decode,
--verify-width 4, mclk pinned 1000 MHz, 2 warmups + 7 reps, matched-clock (sclk floats on this APU, controlled statistically).828186a, #19)fp3-isolated kernel time, type-104 vs the unchanged type-101 fp4-fast control (clock-independent ratio): −18.5% (0.379 → 0.309, rocprofv3
--kernel-trace).Correctness
(*) fp3 VDR=4 is not byte-identical. Regrouping the per-lane accumulation reassociates the cross-lane float reduction (≤ ~1 ULP per block, far below the argmax margin). Validated instead by:
accept_ratepreserved (0.78 → 0.79);fp2 is fully byte-identical (integer accumulation is associative; lane assignment unchanged;
ds_swizzleis bit-identical to the genericwarp_reduce_sumladder). Its +1% is absorbed in the combined build (26.6 == fp3-alone) and is included as a zero-risk companion cleanup.Notes
width == 32); non-HIP and wave64 paths fall back to the generic kernel. No effect on any other quant type or backend.